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A  Framework  for  Categorizing 
Key  Drivers  of  Risk . 

This  technical  report  features  a  systemic 
approach  for  managing  risk  that  takes  into 
account  the  complex  nature  of  distributed 
environments.  By  starting  at  the  top, 
examining  program  objectives  and  then... 
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About  Carol  Woody 


Sr.  Member  of  the  SEI  Technical 
Staff 

Leads  a  team  in  CERT  addressing 
critical  gaps  in  assurance  and 
survivability 

25  years  of  experience  in  software 
management,  acquisition, 
development,  and  implementation 
in  large  complex  organizations 
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Polling  Question  1 


How  did  you  hear  about  this  webinar? 

Email  invitation  from  the  SEI 

•  SEI  Website 

•  Website  with  webinar  calendar  (ie  www.webinar-directory.com) 
Social  Media  site  (  Linkedln,  Twitter) 

•  Other 
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Why  a  Framework 


The  framework  was  developed  to  address  the  following 
research  questions: 

How  can  mission  survivability  be  maintained  as  interoperability  of 
systems  increases? 

Research  sponsored  by  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(OUSD  [AT&L]) 

How  can  operational  impacts  (such  as  information  security)  be  tied 
to  technology  changes  in  operational  mission  execution? 

U.S.  Air  Force’s  Electronic  systems  Center  (ESC)  Cryptologic  Systems  Group  (CPSG) 


Piloted  in  8  complex  operational  environments 
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Framework  Description 
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Purpose  of  the  Framework 


Establish  an  approach  for  assembling  the  broad  range  of  operational 
information  (technology,  people,  and  processes)  to  analyze  it  for 
survivability 

•  Failure  responses  must  be  planned  -  unexpected  usage  (malicious  or 
accidental)  that  drives  operational  execution  outside  of  expected 
behaviors 

•  Survivability  must  be  considered  within  the  context  of  other  quality 
attributes  (performance,  usability,  etc.) 

•  Complexity  and  change  are  unavoidable  so  inconsistencies 
(mismatches)  must  be  assumed  as  we  compose  technology,  people, 
and  processes 

•  Operational  compositions  span  organizational  and  technology 
boundaries 
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Change  is  Always  Occurring 


Survivability  must  accommodate  the  usual  and  the  unexpected 
Usual  problems 

•  Power  and  communication  outages 

•  Snow  storms 

•  Staff  illness  &  death 

Expected  changes 

•  New  technology  insertion 

•  Technology  refreshes 

•  Location  moves 

•  Operational  security 


Catastrophe  is  a  matter  of  scale 
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Polling  Question  2 


What  is  your  organization’s  most  critical  operational  concern? 

a)  Continuity  of  operations 

b)  Natural  disaster  response 

c)  Contractual  compliance 

d)  Vulnerability  management 
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Failure  Model  of  Catastrophe 


Activity 
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An  event  edges 
system  state  closer 
to  Orange  zone. 


EVENT 

A  failure  or 
change 

Increase  in 
work  load 
or  usage 

Reduction 
in  capacity 


A  second  event  followed 
by  increased  usage 
pushes  state  towards  Red. 
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Survivability  Analysis  Framework  (SAF)  - 1 


Framework  for  operational  survivability  analysis 
Process: 

•  Select  a  critical  business  process  for  detail  analysis 

specific  example  of  people  applying  technology  to  perform  an 
operational  mission 

•  Identify  operational  success  criteria 

•  Describe  critical  steps  in  depth 

Sequenced  activities 

Participants 

Resources 

•  Identify  and  analyze  ways  critical  steps  may  not  function  as  intended 
and  opportunities  for  mitigation 
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Survivable  Analysis  Framework  (SAF)  -  2 


Preconditions 


T 

People  Actions  Resources 

T 

Postconditions 


Stresses  Acceptable 

► Outcomes 

Analysis  •  Potential  failure  conditions 

•  Likelihood  of  error  conditions 

•  Impact  of  occurrences 

•  Recovery  strategies 
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People  and  Resources 


People  can  include 

•  direct  performers  of  operational  actions 

•  data  entry,  inquiry,  verification,  audit,  synthesis  among  multiple  information 
sources,  administration  for  technology  components,  authentication  and 
authorization  authorities 

Resources  can  include 

•  hardware — servers,  data  storage  devices,  PCs,  PDAs,  routers,  telephone 
switches,  satellite  relays,  physical  access  controls,  and  similar  devices 

•  software — operating  systems  for  each  hardware  platform,  configuration 
management,  databases,  firewalls,  network  protocols,  packet  switches, 
authentication  packages,  Web  applications,  local  and  remote  procedures 

•  policies  and  practices — certification  and  accreditation,  third-party  access 
management,  outsourcing 

•  contracts,  governance  controls,  and  the  like 
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Polling  Question  3 


Do  you  analyze  the  operational  survivability  of  your  critical  business 
processes  (missions)? 


a)  Yes 

b)  No 
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Example  1 :  Doctor  Orders  Blood  Test 


Simple  example  to  show  the  structure  of  information  used  in  SAF 
Focus  on  failure  analysis 
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Mission  Thread  Examples 
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Example  1 :  Scenario  Steps 


a.  Patient  brought  to  emergency  room  with  chest  pains. 

b.  Dr.  Emergency  reviews  available  records 

c.  Dr.  Emergency  develops  a  treatment  plan. 

d.  Dr.  Emergency  orders  series  of  blood  tests. 

e.  Phlebotomist  from  laboratory  arrives  to  collect  first  specimen  which 
is  sent  to  the  lab. 

f.  Lab  receives  specimen,  performs  centrifuge  and  delivers  serum  to 
appropriate  testing  stations. 

g.  The  Lab  system  notifies  the  ordering  system  that  the  specimen  has 
been  received  fortesting. 
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Example  1 :  Operational  Context 


Doctor  is  using  a  handheld  device  to  review  patient  records,  to  record 
information  during  the  exam,  to  order  tests,  and  to  receive  test 
results. 

The  Lab  is  a  separately  run  business  that  has  contracted  to  provide 
services  to  all  of  the  local  hospitals. 

For  privacy  purposes,  the  Lab  does  not  have  patient-specific 
information.  The  Lab  bills  the  hospital,  and  the  hospital  bills  the 
patient. 
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Example  1 :  Time-line  of  Actions  -  People 


Describe  how  each  person/organization  is  involved  in  each 
step  and  who  has  controlling  authority  (C) 


\v  Action 

People 

A: 

Emergency 

Room 

Encounter 

B:  Doctor 
reviews 
patient 
records 

C:  Doctor 
develops 
treatment 
plan 

D:  Doctor 
orders  blood 
tests 

E: 

Phlebotomist 
arrives  and 
take  specimen 

Patient 

C 

X 

Doctor 

X 

C 

C 

C 

Lab 

Phlebotomist 

C 

Lab 

Technician 
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Example  1 :  Time-line  of  Actions  -  Resources 


Describe  the  resources  involved  in  an  action 


Action 

Resources 

A: 

Emergency 

Room 

Encounter 

B:  Doctor 
reviews  patient 
records 

C:  Doctor 
develops 
treatment  plan 

D:  Doctor 
orders  blood 
test 

Handheld  device 

X 

X 

X 

Hospital 

Communication 

Hub 

X 

X 

X 

Hospital 

Admissions 

X 

Hospital  medical 
'ecords 

X 

X 

Hospital  Lab 
communications 

X 

_ab  Systems 

X 

r\ 

'CERT 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


25 


Example  1 :  Describing  a  Critical  Step 


Step  D 


Doctor  orders  blood  tests 


Precondition 


Treatment  plan  defined 

Required  lab  tests  identified 

Handheld  connectivity  to  Lab 

Dr.  E.  authorization  to  order  tests  for 
this  patient 


Action 


Enter  order  on  handheld 

Order  accepted,  verified,  and 
accepted  by  Lab 


Post-Condition 


Test  confirmed  and  scheduled 
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Example  1 :  Failure  Outcomes  for  Step  D 


Missing  (or  delayed)  results: 

•  some  or  all  tests  are  not  done 

Wrong  results: 

•  some  unrequested  tests  were  performed 

•  results  do  not  reflect  the  actual  sample 

Disclosure: 

•  results  are  disclosed  to  unauthorized  person 

•  test  results  are  not  associated  with  the  correct  patient 

•  test  results  are  not  associated  with  the  correct  doctor 
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Example  1 :  Causes  of  Failure 


Failure:  Missing  test  results 

•  Paperwork  requiring  tests  to  be  run  was  lost  or  misplaced 

•  Blood  samples  were  lost,  contaminated,  or  misplaced 

•  Some  tests  were  not  run  by  the  technician 

•  Wrong  tests  were  run  by  the  technician 

•  Some  or  all  test  results  were  not  associated  with  the  correct  patient  by 
the  lab 

•  Some  or  all  test  results  were  not  associated  with  the  right  doctor  by  the 
lab 

•  Testing  machine  did  not  produce  results 

•  Testing  machine  was  not  working  and  could  not  produce  results 
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Example  2:  Joint  Military  Operation 


Complex  example  to  show  how  the  SAF  structuring  of  information 
supports  analysis 

Focus  on  the  impact  of  technology  change  on  operational  survivability 
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Example  2:  Military  Joint  Forces  Mission  Thread 


An  army  unit  on  patrol  spots  a  missile  launcher  preparing  to  fire.  The  unit  calls 
their  commander  and  provides  a  description  of  the  launcher  and  its  location. 
Even  though  the  launcher  is  in  the  Army’s  area  of  responsibility,  in  this 
scenario  the  Army  does  not  have  an  appropriate  weapon  to  bring  to  bear  (for 
example,  the  artillery  could  be  in  use  on  other  targets).  However,  the  Air  Force 
has  a  suitable  platform  and  is  tasked  as  Executive  Agent  to  further  prosecute 
or  strike  of  the  target.  The  Army  remains  the  authority  for  the  strike  even 
though  the  Air  Force  will  perform  the  engagement. 
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Example  2:  Scenario  Steps 


1.  Army  unit  sees  “something”  (e.g.  Missile  launcher  preparing  to  fire.) 

2.  Unit  calls  their  command  post  via  satellite  connection  and  provides  a 
description  of  the  launcher  and  its  location. 

3.  UAV  is  moved  into  position  to  evaluate  the  potential  target. 

4.  Army  Command  notifies  Joint  Services  of  a  critical  target  requesting  approval 
and  support 

5.  Joint  Services  approves  the  target  for  critical  support 

6.  Joint  Services  review  weapon(s)  and  delivery  platform  options  based  on 
timing,  collateral  damage,  desired  effect,  etc. 

7.  Air  Force  plane  selected  to  attack  target 

8.  Order  transmitted  to  pilot  (usually  verbally) 

9.  Pilot  executes  attack  coordinated  by  JTAC  (ground  resource  embedded  in 
Army  unit) 

10.  Multiple  assessments  lead  to  unified  battle  damage  assessment  (BDA) 
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Example  2:  Operational  Context 


Mixture  of  communication  mechanisms  with  high  degradation  potential 

Wide  spectrum  of  failure  potentials  (battle  operations) 

Wider  spectrum  of  stakeholders  (with  potentially  conflicting  needs) 

Multiple  systems  and  their  interactions  (each  participant  is  independent 
operator  participating  in  system  of  systems) 
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Example  2:  Mission  Interactions 
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Example  2:  Describing  a  Critical  Step 


Step  9  Target  Attack 

Communication  between  JTAC  and 
aircraft  established 

Communication  between  JTAC  and  Army 

Precondition  Target,  friendlies,  etc  are  “marked” 

appropriately 

Army  approval:  The  supported  army 
ground  unit  provides  approval  to  the 
JTAC  to  release  the  weapons. 

Synchronization  of  target  identification 

Action  JTAC  provides  target  corrections  to 

aircrews  as  needed 

JTAC  clears  or  aborts  aircraft  to  attack 
Post-Condition  Target  is  attacked 
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Example  2:  Mission  Critical  Resource 


Evaluate  changes  to  a  mission  critical  resource  in  Step  9 

•  JTAC  and  Aircraft  establish  secure  communications 

Potential  resource  context  changes  over  time 

•  Currently  voice,  line  of  site  (LOS) 

•  Enhanced  to  text  and  voice,  LOS  with  satellite  backup 

•  Expanded  to  satellite  image  sharing  from  UAV  to  JTAC  and  JTAC  to 
aircraft 
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Example  2:  Critical  Resource  Impact  Evaluation 


Failure  impact  potential: 

•  High:  mission  abort,  mission  errors  with  fratricide,  wrong  target 

•  Medium:  mission  delays;  insufficient  attack  power;  loss  of  IA  for 
mission  (exposure) 

•  Low:  future  mission  potential 

Selected  failures  for  JTAC  to  aircraft  communication  (changes) 

•  Failure  of  communication  connection  -  high  impact  (reduced  with 
satellite  backup) 

•  Partial  transmission  (retry  required)  -  medium  impact  (increases  with 
more  channels  -  using  more  bandwidth) 

•  Encryption  failure  for  communication  -  medium  impact  (unchanged) 

•  Incompatible  communication  software  -  medium  to  high  impact 
(increases) 
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Example  2:  Failure  Analysis 


For  each  failure  option: 

•  Outcome  for  resource  if  failure  realized  (disclosure,  modification, 
loss/destruction,  unavailable) 

•  Impact  on  mission  of  compromised  resource 

•  Impact  rating  (high,  medium,  low) 

•  Mitigations 

•  Response  strategy  (accept,  monitor,  mitigate) 

Changes  to  system  and  software  requirements  resulting  from  decisions 
to  mitigate 
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Polling  Question  4 


Do  you  consider  the  potential  of  operational  failure  when  evaluating 
technology  changes? 


a)  Yes 

b)  No 
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Multi-View  Decision  Making  (MVDM) 


Provides  an  approach  that  addresses  the  breadth  and  depth  of 
activities,  decisions,  and  products  that  must  come  together  to 
successfully  address  complex  software  development 

Evaluate  and  compare  the  mission,  the  integrated  operational 
execution,  and  acquisition 

•  Mission-Oriented  Success  Analysis  and  Improvement  Criteria 
(MOSAIC) 

•  Survivability  Analysis  Framework  (SAF) 

•  System  of  Systems  Interoperable  Acquisition 
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Using  SAF  with  Other 
Analysis  Techniques 
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Security  Assurance  Case 


Structured  view  (claims,  arguments,  and  evidence)  that  describes  how 
potential  security  failures  are  addressed 

•  Connects  mitigations  with  the  threats  they  are  addressing 

•  Identifies  assurance  gaps  (missing  evidence  of  mitigations) 

Example  Claim:  Supplier  follows  suitable  security  coding  practices 

•  Sub-claim:  Supplier  bans  and  enforces  the  ban  on  use  of  dangerous 
application  programming  interfaces 

•  Sub-claim:  Static  analysis  tools  with  appropriate  vulnerability  coverage 
are  applied  at  appropriate  times  throughout  development 

Evidence:  Percentage  of  current  code  subject  to  static  analysis 

Evidence:  Listing  of  applicable  coding  weaknesses 

Evidence:  Percentage  of  applicable  coding  weaknesses  covered 
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Assurance  Case  Structure 


A:  Assumptions  fdiT 
Claim 


F-:r  purposes -ol  this- dam 
we  assume  ... 


Cp  Survivable  system 

The  system  urder 
constsersliofl  rs  Survivab'e 

I 


5:  Hdi-jr  Ps  Lo 
Pns  cessing 


Ar^ue  over  the  hoards 
to  pro  cess  in  9 

- 1— - ' 


Ctxi:  5 v a- Li! m 

Document  X  rte  scribes 
U  ie  systems  anc  its. 
prcpen-ss  related  te 
survivability 


/c 
I  s 

k 


Cist:  Hazards  :o 
SuHvfvaL  li!y 

Hazarc  1.  Hazard 
2 


r\ 

'CERT 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


42 


Summary 
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Use  SAF  to  Plan  for  Survivability 


Plan  for  normal  operation  and  failure  response 

•  Develop  range  of  mission  threads  that  are  critical  for  normal  operations 

•  Identify  critical  mission  activities,  participants,  and  resources 

•  Evaluate  the  range  of  unexpected  behaviors  that  could  contribute  to 
mission  failure 

•  Mitigate  potential  failure 

Plan  for  operational  mission  impact  of  technology  changes 

•  Identify  critical  mission  resources  affected  by  proposed  changes 

•  Evaluate  the  operational  impact  of  failures  in  the  current  and  changed 
operational  context 

•  Identify  change  requirements  based  on  mitigations  for  survivability 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE 
ENGINEERING  INSTITUTE  MATERIAL  IS  FURNISHED  ON  AN  “AS-IS"  BASIS. 
CARNEGIE  MELLON  UNIVERSITY  MAKES  NO  WARRANTIES  OF  ANY  KIND, 
EITHER  EXPRESSED  OR  IMPLIED,  AS  TO  ANY  MATTER  INCLUDING,  BUT 
NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR  PURPOSE  OR 
MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM  USE  OF 
THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY 
WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT, 
TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 

Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to 
infringe  on  the  rights  of  the  trademark  holder. 

This  Presentation  may  be  reproduced  in  its  entirety,  without  modification, 
and  freely  distributed  in  written  or  electronic  form  without  requesting  formal 
permission.  Permission  is  required  for  any  other  use.  Requests  for  permission 
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permission(a)sei.cmu.edu. 
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